[CIVIS-11461] Add example with environment variables compatible with civis studios #42
Conversation
gskifstad
force-pushed
the
CIVIS-11461-add-code-cloud-compatible-dbt-example
branch
from
9de2dfb to
13ede44
Compare
|
todo: check again the compatibility with studios. it seems to be working in the dbt script now! https://platform.civisanalytics.com/spa/#/scripts/dbt/330758402 |
gskifstad
changed the title
| # The following can be used in both studios and Platform dbt scripts, assuming | ||
| # parameters* like the following exist in the studio: | ||
| # "params": [ | ||
| # { | ||
| # "name": "civis_consumers", // a redshift database parameter | ||
| # "description": null, | ||
| # "type": "database", | ||
| # "value": "{\"credential\":1234,\"database\":5678}" | ||
| # }, { | ||
| # "name": "dbt_schema", | ||
| # "description": null, | ||
| # "type": "string", | ||
| # "value": "my_dbt_schema" | ||
| # }, { | ||
| # "name": "studio_redshift_prefix", | ||
| # "description": null, | ||
| # "type": "string", | ||
| # "value": "civis_consumers" // matches the parameter name for the redshift database | ||
| # }, { | ||
| # "name": "civis_international_consumers", // a bigquery database parameter | ||
| # "description": null, | ||
| # "type": "database", | ||
| # "value": "{\"credential\":333,\"database\":999}" | ||
| # }, { | ||
| # "name": "studio_bigquery_prefix", | ||
| # "description": null, | ||
| # "type": "string", | ||
| # "value": "civis_international_consumers" // matches the parameter name for the BQ database | ||
| # } | ||
| # ] | ||
| # *Note: this example includes parameters for both redshift and bigquery, but you'll likely only need | ||
| # one or the other | ||
| studio-or-dbt-script: | ||
| target: redshift_adapter # sets the default target | ||
| outputs: | ||
| # Works for redshift or postgres | ||
| redshift_adapter: | ||
| type: "{{ env_var(env_var('STUDIO_REDSHIFT_PREFIX', 'DATABASE') ~ '_TYPE') }}" | ||
| host: "{{ env_var(env_var('STUDIO_REDSHIFT_PREFIX', 'DATABASE') ~ '_HOST') }}" | ||
| port: "{{ env_var(env_var('STUDIO_REDSHIFT_PREFIX', 'DATABASE') ~ '_PORT') | as_number }}" | ||
| user: "{% if env_var('STUDIO_REDSHIFT_PREFIX', '') %}{{ env_var(env_var('STUDIO_REDSHIFT_PREFIX') ~ '_CREDENTIAL_USERNAME') }}{% else %}{{ env_var('DATABASE_USERNAME') }}{% endif %}" | ||
| pass: "{% if env_var('STUDIO_REDSHIFT_PREFIX', '') %}{{ env_var('DBT_ENV_SECRET_' ~ env_var('STUDIO_REDSHIFT_PREFIX') ~ '_CREDENTIAL_PASSWORD') }}{% else %}{{ env_var('DBT_ENV_SECRET_DATABASE_PASSWORD') }}{% endif %}" | ||
| dbname: "{% if env_var('STUDIO_REDSHIFT_PREFIX', '') %}{{ env_var(env_var('STUDIO_REDSHIFT_PREFIX') ~ '_DATABASE') }}{% else %}{{ env_var('DATABASE_NAME') }}{% endif %}" | ||
| schema: "{{ env_var('DBT_SCHEMA') }}" | ||
| threads: 4 |
There was a problem hiding this comment.
this profile works for redshift in studios and platform dbt scripts! but im not sure if it is worth publishing? it is a bit complicated, but could be helpful.... I'd be curious if others find it readable or not.
options:
- move it into its own profiles file, so it is easier to ignore?
- remove it from platform-code-examples entirely
- keep it in platform-code-examples until we handle https://civisanalytics.atlassian.net/browse/CIVIS-11950
cc @leanne73 do you have thoughts on this?
There was a problem hiding this comment.
I can read it on my big monitor, but I was struggling to comprehend it from my laptop's smaller screen when looking at it last week lol.
I'm leaning towards moving both the studio examples to a separate profiles file? But after 11950, I think we'd only need the first example, which seems reasonable to keep in the same place as everything else...
lemme check with fessler and see if it looks like we'll be prioritizing CIVIS-11950 into an upcoming sprint. If so, then I think we just keep the simpler profile. Otherwise, it probably makes sense to keep them both, but maybe in a separate file.
2 participants
No description provided.